<html>
<head><meta charset="utf-8"><title>Testing cargo clippy · clippy · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/index.html">clippy</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html">Testing cargo clippy</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="241400078"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241400078" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241400078">(Jun 03 2021 at 14:41)</a>:</h4>
<p>I'm wondering how clippy tests the output of cargo using <code>compiletest</code>. I looked at <a href="https://github.com/rust-lang/rust-clippy/blob/5f746a1c2ae130ef84704c941b962075af6693ac/tests/compile-test.rs#L160"><code>run_ui_cargo</code> in <code>compile-test.rs</code></a> but I can't find anything that tells <code>compiletest</code> to run <code>cargo</code> instead of just the clippy driver. The code seems to directly run the clippy driver on the <code>src/main.rs</code> of the test crates, but by doing so I would expect that the dependencies are not compiled. What am I missing? We would like to reuse the same mechanism to test <a href="https://github.com/viperproject/prusti-dev"><code>cargo prusti</code></a> and I guess that in general other cargo commands could do the same.</p>



<a name="241401483"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241401483" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241401483">(Jun 03 2021 at 14:50)</a>:</h4>
<p>note that whatever the mechanism is, it's buggy: <a href="https://github.com/rust-lang/rust/issues/76495#issuecomment-852827386">https://github.com/rust-lang/rust/issues/76495#issuecomment-852827386</a></p>



<a name="241401524"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241401524" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241401524">(Jun 03 2021 at 14:50)</a>:</h4>
<p>I would look at how the in-tree compiletest does it instead</p>



<a name="241402953"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241402953" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241402953">(Jun 03 2021 at 15:00)</a>:</h4>
<p><span class="user-mention silent" data-user-id="232545">Joshua Nelson</span> <a href="#narrow/stream/257328-clippy/topic/Testing.20cargo.20clippy/near/241401483">said</a>:</p>
<blockquote>
<p>note that whatever the mechanism is, it's buggy: <a href="https://github.com/rust-lang/rust/issues/76495#issuecomment-852827386">https://github.com/rust-lang/rust/issues/76495#issuecomment-852827386</a></p>
</blockquote>
<p>Uh, thanks for pointing that out. Maybe <code>rustwide</code> is still the best way to test cargo commands.</p>



<a name="241404593"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241404593" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241404593">(Jun 03 2021 at 15:09)</a>:</h4>
<p>rustwide is for sandboxing. I'm not sure how it's related here.</p>



<a name="241404646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241404646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241404646">(Jun 03 2021 at 15:10)</a>:</h4>
<p>what's wrong with in-tree compiletest?</p>



<a name="241406959"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241406959" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241406959">(Jun 03 2021 at 15:25)</a>:</h4>
<p>It might be an overkill, but rustwide would allow to run a command in an environment with a fresh target folder, which seems related to the issue that you linked.</p>



<a name="241407028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241407028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241407028">(Jun 03 2021 at 15:25)</a>:</h4>
<p>Is the in-tree compiletest exposed as a library, usable from crates that are not in-tree?</p>



<a name="241408201"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241408201" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241408201">(Jun 03 2021 at 15:33)</a>:</h4>
<p>One more limitation of the in-tree compiletest is that the <code>CompileFail</code> mode has been removed, while <code>compiletest_rs</code> still supports it. The <code>//~ ERROR</code> syntax is very useful to annotate test cases in our tool.</p>



<a name="241408851"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241408851" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241408851">(Jun 03 2021 at 15:37)</a>:</h4>
<p>Anyway, I'm still wondering how clippy manages to check cargo's output using compiletest (even if it's buggy).</p>



<a name="241409212"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241409212" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241409212">(Jun 03 2021 at 15:40)</a>:</h4>
<p>The in-tree cargotest looks interesting, but it doesn't check for expected errors.</p>



<a name="241410249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241410249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241410249">(Jun 03 2021 at 15:47)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116427">Federico Poli</span> <a href="#narrow/stream/257328-clippy/topic/Testing.20cargo.20clippy/near/241406959">said</a>:</p>
<blockquote>
<p>It might be an overkill, but rustwide would allow to run a command in an environment with a fresh target folder, which seems related to the issue that you linked.</p>
</blockquote>
<p>at that point it would be simpler to just fix the bug in compiletest</p>



<a name="241410324"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241410324" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241410324">(Jun 03 2021 at 15:48)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116427">Federico Poli</span> <a href="#narrow/stream/257328-clippy/topic/Testing.20cargo.20clippy/near/241408201">said</a>:</p>
<blockquote>
<p>One more limitation of the in-tree compiletest is that the <code>CompileFail</code> mode has been removed, while <code>compiletest_rs</code> still supports it. The <code>//~ ERROR</code> syntax is very useful to annotate test cases in our tool.</p>
</blockquote>
<p>I don't understand what you mean. Rust-lang/rust has tons of tests with ERROR annotations.</p>



<a name="241413095"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241413095" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241413095">(Jun 03 2021 at 16:08)</a>:</h4>
<p>Thanks for pointing that out. I thought that <code>CompileFail</code> was the only mode checking for ERROR annotations.</p>



<a name="241414412"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/241414412" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#241414412">(Jun 03 2021 at 16:17)</a>:</h4>
<p>Indeed, by looking at the implementation of <a href="https://github.com/Manishearth/compiletest-rs/blob/37f4f9cd409cac902bf99c99e467b855a1187326/src/runtest.rs#L2291"><code>run_ui_test</code></a> <code>compiletest_rs</code> does not check ERROR annotations for UI tests, but the compiletest in rust-lang/rust <a href="https://github.com/rust-lang/rust/blob/835150e70288535bc57bb624792229b9dc94991d/src/tools/compiletest/src/runtest.rs#L3394">does</a>. That's why I was confused.</p>



<a name="242182933"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/Testing%20cargo%20clippy/near/242182933" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Federico Poli <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/Testing.20cargo.20clippy.html#242182933">(Jun 10 2021 at 09:58)</a>:</h4>
<p>The answer to my initial question is that the <code>run_ui_cargo</code> test doesn't actually run <code>cargo</code>. It runs <code>rustc</code> with some special flags to handle dependencies, faking a <code>cargo</code> run. See <a href="https://github.com/rust-lang/rust-clippy/blob/f7d09b45e9dbb32c5524d5f3b0838401c848bbf2/tests/compile-test.rs#L79-L84">here</a>.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>